Knowledge distillation methods for efficient unsupervised adaptation across multiple domains

Beyond the complexity of CNNs that require training on large annotated datasets, domain shift between design and operational data has limited adoption in many real-world applications. For instance, person re-identification, videos are captured over a distributed set cameras with non-overlapping viewpoints. The source (e.g. lab setting) target cameras) domains may lead to significant decline recognition accuracy. Additionally, state-of-the-art not be suitable for such real-time applications given their computational requirements. Although several techniques have recently been proposed address problems through unsupervised adaptation (UDA), or accelerate/compress knowledge distillation (KD), we seek simultaneously adapt compress generalize well across multiple domains. In this paper, propose progressive KD approach single-target DA (STDA) multi-target (MTDA) CNNs. Our method KD-STDA adapts CNN single by distilling from larger teacher CNN, trained both order maintain its consistency common representation. This is extended MTDA problems, where teachers used distill student CNN. A different assigned each model UDA, they alternatively preserve specificity target, instead directly combining using fusion methods. compared against methods compression STDA Office31 ImageClef-DA image classification datasets. It also Digits, Office31, OfficeHome. settings – KD-MTDA results indicate our can achieve highest level accuracy domains, while requiring comparable lower complexity.